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FOREWORD 


This  project  was  conducted  in  support  of  a Commander,  Naval  Air 
Systems  Command  (AIR-413)  request  to  furnish  instructional  technology 
consulting  support  to  the  F-14  Aircrew  Task  Analysis  and  Training  Inte- 
gration Program. 

The  entire  staff  of  VF-124,  Miramar  NAS,  should  be  recognized  for 
their  support  of  this  effort.  Within  that  organization,  LCDR  C.  Flack  Logan, 
LCDR  Jim  Lang,  LCDR  Joel  Grafman,  and  LT  Hoot  Gibson  deserve  special  men- 
tion for  their  cooperation.  Mr.  Mike  Plunkett  of  COMNAVAIRSYSCOM  (AIR-4132D) 
has  been  consistently  helpful  and  supportive.  The  San  Diego  staff  of  Veda, 
Incorporated,  particularly  Mr.  George  Viglotti,  Mr.  Thel  Hookes,  and 
Ms.  Nancy  Marcuse,  have  contributed  significantly  to  the  success  of  the 
program.  Mr.  John  Moore,  also  of  Veda,  has  made  unique  and  significant 
Inputs  to  both  the  project  itself  and  to  the  present  document. 


J.  J.  CLARK IN 
Commanding  Officer 
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SUMMARY 

Problem 

Current  Navy  training  in  the  flight  and  operation  of  state-of-the-art 
weapons  systems,  although  adequate,  falls  short  of  what  la  possible  through 
modern  instructional  technology. 

Objective 

The  purposes  of  this  effort  were  (1)  to  develop  a systematic  approach 
to  the  design  of  Instruction  for  operators  of  sophisticated  weapons  systems 
and  (2)  to  produce  effective  training  syllabi  for  the  aircrew  of  the  F-14 
fighter  aircraft. 

Approach 

Specific  instructional  techniques  and  models  were  applied  to  the  design 
of  a revised  P-14  aircrew  training  program.  No  particular  school  of  thought 
or  technique  was  favored.  In  the  case  of  categorizing  operator  tasks,  select- 
ing training  media,  and  sequencing  of  instruction,  innovative  and  original 
models  and  techniques  were  developed  specifically  for  the  program. 

Findings 

This  process  is  not  the  long-sought-after,  fully  proceduralized  instruc- 
tional design  system.  Many  of  the  techniques  developed  came  strictly  out 
of  necessity,  and  many  have  been  validated  only  to  the  extent  that  they 
allowed  the  process  to  continue.  An  extensive  review  of  the  state-of-the- 
art  of  instructional  technology  has  produced  few  hard  findings,  but  rether 
a number  of  significant  guidelines  for  the  designer  of  instruction. 

Conclusions 


1.  Early  results  of  the  current  program  Indicate  that  the  described 
process  is  producing  an  effective,  job-relevant  training  system. 

2.  The  documentation  produced  by  the  ISD  contractor  will  allow  rapid 
integration  of  the  new  syllabi  and  a new  system  for  computer-managed 
instruction. 

Recommendations 

1.  The  F-14  aircrew  training  syllabi  should  be  implemented  with  minimum 
disruption  of  ongoing  training. 


2.  The  algorithms  sad  decision  aids  developed  through  this  program 
should  become  part  of  the  Navy's  available  instructional  technology  re- 
sources. 

3.  The  possibility  of  using  part-task  trainers  in  the  F-14  training 
program  should  be  thoroughly  investigated. 


vil 
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4.  The  CAI  options  resulting  from  the  nedia  selection  algorithms 
should  be  carefully  Investigated. 

5.  The  R&D  community  should  further  study  the  hlerarchlal  nature  of 
learning. 
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Problem 

Current  Navy  training  in  the  flight  and  operation  of  state-of-the-art 
weapons  systems,  although  adequate,  falls  short  of  what  is  possible  through 
modern  instructional  technology. 

Purpose 

The  purposes  of  this  research  were  (1)  to  develop  a systematic  approach 
to  the  design  of  instruction  for  operators  of  sophisticated  weapons  systems 
and  (2)  to  produce  effective  training  syllabi  for  the  aircrew  of  the  F-14 
fighter  aircraft.  This  report  presents  one  way  to  design  job-relevant 
training  and  describes  the  process  by  which  F-14  training  syllabi  have 
been  revised.  Specific  prescriptions  for  instructional  sequencing  and 
categorizing  behavior  represent  Innovations  (actually  instructional  aids) 
that  are  quantum  jumps  forward  in  Instructional  technology. 

Background 

Instructional  Systems  Design 

The  Navy  is  committed  to  the  systematic  design  of  training  courses 
(Chief  of  Naval  Education  and  Training,  1975;  Scanland,  1974).  The  approach, 
which  is  called  variously  course  design  procedure  (CDP),  systems  approach 
to  training  (SAT),  and  instructional  systems  design  (ISD) , essentially  takes 
the  training  course  developer  from  a set  of  job  tasks  through  the  develop- 
ment of  behavioral  objectives  to  the  conduct  of  a job-relevant  training  pro- 
gram. Montemerlo  and  Tennyson  (1976)  have  traced  the  history  of  what  is 
now  called  ISD  from  its  beginnings  in  the  systems  analysis  technology  in 
the  1940s  (Kershaw  & McKean,  1958;  Hoehn,  1960)  to  its  current  status  as 
the  way  to  design  training  (Butler,  1967;  Rundquist,  1970;  Chief  of  Naval 
Education  and  Training,  1975).  Readers  interested  in  a complete  history 
and  bibliography  of  SAT  are  referred  to  Montemerlo  and  Tennyson  (1976). 

That  history  describes  the  current  research  and  development  issues 

in  ISD: 

The  fundamental  issue  requiring  resolution  is  the  nature 
of  SAT  (Systems  Approach  to  Training).  The  original 
concept  holds  that  program  development  cannot  be  pro- 
ceduralized,  and  that  it  is  therefore  the  proper  domain 
of  the  expert  training  program  designer.  The  second  SAT 
concept  is  that  training  program  design  is  proceduralized, 
and  that  manuals  can  be  developed  which  are  used  by  per- 
sonnel less  competent  and  less  costly  than  the  expert  to 
produce  equally  effective  training  programs.  The  resolu- 
tion of  this  issue  created  by  these  two  conflicting  con- 
cepts probably  lies  in  their  synthesis.  The  fully  proce- 
duralized SAT  concept  has  no  basis  in  existing  psychological 








theory  and  research.  Vet  experience  with  it  has  shown 
that  laymen  can  be  productive  in  some  aspects  of  training 
program  design.  The  original  SAT  concept,  which  relies  on 
the  expert,  provides  no  information  on  the  particular 
skills  necessary  to  qualify  one  as  an  expert.  Hard  data 
are  needed  to  determine  what  skills  are  necessary  and  the 
degree  of  proficiency  to  which  each  is  required  for  the 
accomplishment  of  the  various  steps  of  training  program 
development.  (Montemerlo  & Tennyson,  1976,  p.  13) 

The  question  of  whether  or  not  an  ISD  cookbook  for  the  training 
novice  is  a reasonable  goal  is  unresolved.  However,  the  requirement  for 
efficient  and  effective  training  programs  remains.  To  the  operational  fleet, 
the  R&D  issue  is  secondary  to  the  critical  demand  for  good  training  right 
now.  This  real-world  constraint  is  undoubtedly  a major  cause  of  the  lack 
of  hard  data  supporting  either  an  ISD  cookbook  for  the  "job  expert  but 
training  novice"  or  a set  of  ISD  guidelines  for  the  "instructional  design 
expert . " 

The  Navy  Personnel  Research  and  Development  Center  (NAVPERSRANDCEN) 
and  its  predecessors  have  been  involved  in  ISD-related  research  for  more  than 
a decade  (Curran,  1966;  Rundquist,  1970;  Brock,  DeLong,  & McMichael,  1975) 
and  aircrew  training  design  for  the  last  5 years  (e.g.,  Brock,  1976;  McLachlan  & 
Crawford,  1975;.  This  research  has  primarily  been  the  result  of  targets  of 
opportunity  rather  than  the  result  of  a cohesive,  long-term  R&D  program. 

In  addition,  as  NAVPERSRANDCEN  has  conducted  general  training  research, 
serendipitous  inputs  to  an  ISD  model  have  occurred. 

What  has  evolved  is  a flexible,  dynamic  model  for  the  design  of 
job-relevant  training.  This  report  will  document  the  current  status  of 
that  model. 

F-14  Aircrew  Training 

Fighter  Squadron  124  (VF-124)  is  the  Pacific  Fleet  training  squadron 
for  the  F-14  Tomcat  fighter  aircraft.  The  F-14  is  the  Navy's  newest  fighter 
aircraft  and  is  generally  conceded  to  represent  the  state-of-the-art  in 
airborne  weapons  systems.  Its  aircrew  consists  of  a pilot  and  a naval 
flight  offier  (MFO) . In  a gross  (but  explanatory)  oversimplification, 
the  pilot  flies  the  plane  and  the  NFO  operates  the  weapons  systems.  The 
NFO  has  no  flight  control  instrumentation  in  his  cockpit,  and  the  pilot 
has  few  weapon  system  data  displays  in  his.  There  is  a minimum  overlap 
of  inflight  tasks.  (For  a general  description  of  the  F-14,  see  Stevenson, 

1975). 

Before  the  initiation  of  the  work  reported  herein,  the  history  of 
the  VF-124  training  program  was  fairly  typical:  Initial  training  design 

was  performed  by  the  weapons  system  prime  contractor  with  assistance  from 
the  Navy  test  pilot/NFO  community.  Before  the  F-14  was  operational,  a cadre 
of  Navy  figher  aircrew  personnel  was  trained  in  F-14  operation  by  the  prime 
contractor.  This  cadre  then  designed  the  VF-124  aircrew  training  syllabi, 
which  have  continued  to  evolve  under  the  guidance  of  an  ISD  department  within 
the  training  squadron. 


The  design  process  for  the  initial  syllabi  began  with  the  functions 
and  characteristics  of  the  F-14  weapons  system.  The  emphasis  of  the  current 
program  is  to  train  about  the  F-14  rather  than  how  to  operate  it. 

Commander,  Navy  Air  Systems  Command  (COMNAVAIRSYSCOM)  therefore 
authorized  a training  program  redesign  effort  directed  at  the  VF-124  syllabi. 
Veda,  Incorporated,  at  San  Diego,  California,  was  selected  as  the  contractor 
for  this  effort.  NAVPERSRANDCEN  was  requested  to  serve  as  instructional 
advisor  to  the  program.  One  of  the  factors  in  COMNAVAIRSYSCOM' s decision 
was  that  one  of  the  weapons  system  subcontractors  was  developing  a computer- 
managed  training  (CMT)  system  for  VF-124;  it  was  hoped  that  the  CMT  and  the 
ISD  efforts  would  dovetail  into  an  efficient  and  effective  training  program. 

VF-124  trains  its  aircrew  in  modern  classrooms,  learning  carrels 
equipped  with  both  tape/slide  and  video  tape  playback  capabilities,  three 
operational  flight  trainers  (OFT),  three  mission  trainers,  a cockpit  pro- 
cedures trainer,  one  AWG-9  operational  maintalnance  trainer,1  and  an  F-14 
cockpit/ejection  seat  trainer. 


1The  AWG-9  is  the  heart  of  the  F-14  weapons  system.  It  is  made  up  of 
• radar,  computer,  and  interface  system  with  weapons  and  their  associated 
displays  (NATOPS,  Note  1). 
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ANALYSIS  PROCEDURES 


ISD  Model 


The  model  developed  and  applied  by  Veda  and  NAVPERSRANDCEN  in  the 
F-14  ISD  effort  stems  from  that  developed  by  Rundquist  (1970),  as  modified 
by  Brock  and  DeLong  (1975).  The  model  has  also  been  influenced  by  Gagne 
(1970),  Merrill  (1971a),  and  Markel  and  Tiemann  (1973).  The  model  consists 
of  the  following  steps  in  training  course  design: 


1.  Develop  course  mission  (overall  training  goal). 

2.  Identify  job  tasks. 

3.  Establish  job-entry  standards. 

4.  Group  tasks  for  instructional  planning. 

5.  Develop  training  tasks  (job  task  and  school  conditions). 

6.  Specify  the  test  (add  standards  to  training  tasks). 

7.  Complete  the  objectives. 

8.  Organize  the  training  by  objectives  (strategy). 

9.  Develop  media,  job  aids,  criterion  tests,  and  other  course 
material. 

10.  Conduct  the  course  for  validation. 


The  model  also  flows  from  a synthesis  of  NAVPERSRANDCEN’s  expertise  in 
instructional  technology  and  the  contractor's  expertise  in  both  systems 
analysis  and  fighter  aircraft  weapons  systems.  Since  the  model  draws  on 
such  a variety  of  resources  without  advocating  any  specific  school,  it  may 
be  termed  eclectic. 


Course  Mission 


The  mission  of  VF-124  is: 

To  provide  the  knowledge  and  skills  necessary  for  preparing 
the  Category  I replacement  pilot  (RP)  and  the  replacement 
Naval  Flight  Officer  (RN)  for  an  F-14  fleet  assignment  where 
he  will  be  able  to  perform  proficiently  in  all  aspects  of 
aircraft  operation,  weapons  utilization,  and  mission  accom- 
plishment (Fighter  Squadron  124,  1974). 

Task  Analysis2 


Montemerlo  and  Tennyson  (1976)  state:  "There  is  universal  agreement 

in  the  SAT  [systems  approach  to  training]  literature  that  task  analysis 
is  necessary.  However,  there  is  little  agreement  as  to  what  a task  analysis 
is"  (p.  15). 


No  attempt  is  made  in  this  report  to  develop  the  ultimate  definition 
of  task  analysis.  The  task  analysis  method  used  in  this  work  was  a systematic 


2 Much  of  the  description  of  the  F-14  task  analytic  process  is  taken 
from  the  F-14  Pilot  and  NFO  task  inventories  prepared  and  published  for 
the  Navy  by  Veda,  Inc.  (Note  2). 
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process  that  resulted  in  precise  specifications  for  all  tasks,  whether 
cognitive  or  psychomotor,  required  by  the  operators  to  direct  the  weapon 
system  to  the  completion  of  its  mission. 

A review  of  F-14  weapons  system  documentation  was  the  first  step  in 
analyzing  each  task.  Considerable  data  on  aircrew  tasking  exist  in  the 
form  of  operating  manuals,  training  manuals,  system  descriptions,  engineer- 
ing reports,  test  and  evaluation  reports,  and  fleet  operational  reports. 

After  these  data  were  collected  and  organized,  individual  and  group 
interviews  became  a key  source  of  information  and  task  validation.  Job 
experts  (i.e.,  F-14  pilots  and  HFOs)  were  interviewed  in  1-hour  sessions 
by  researchers  responsible  for  developing  the  task  analyses.  During  the 
Interviews,  Veda  analysts  explored  specific  task  areas  with  the  interviewees. 
These  job  experts  described  the  tasks  they  performed  on  varying  missions 
and  with  varying  aircraft  configurations,  and  summarized  their  experiences 
(Table  1).  The  analysts  interviewed  at  least  three  experts  for  each  job. 
Additionally,  analysts  witnessed  a limited  number  of  actual  tasks  such 
as  an  aircrew  performing  a preflight  inspection. 


Table  1 

Average  F-14  Aircrew  Interviewee  Experience 


Item 

Average  Response 

Years  of  military  service 

10 

Years  of  flight  service 

9 

Total  flight  hours 

2068 

F-14  flight  hours 

175 

Number  of  carrier  deployments 

2.4 

Number  of  combat  deployments 

2.2 

Number  of  carrier  landings 

343 

Number  of  F-14  carrier  landings 

5 

Months  of  instructor  experience 

29 

Months  of  F-14  instructor  experience 

10.4 

Note:  These  data  are  taken  from  Veda,  Note  3. 


From  these  procedures  (data  organization,  interviews,  observations), 
the  F-14  aircrew  task  analyses  were  developed.  What  follows  is  a descrip- 
tion of  the  end  products  of  these  analyses  and  their  underlying  assumptions. 

The  basic  mission  of  the  F-14  was  broken  down  into  a hierarchy  of  mission 
phases,  mission  elements,  and  first-order  tasks  (Figure  1). 
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Figure  1.  First. -order  casks — configure  for  attack  (Veda,  Note  2) 


Mission  Phases 


The  central  theme  of  this  hierarchy  development  is  based  on  the 
F-14's  mission  to  augment  the  Navy's  offensive  and  defensive  posture,  with 
special  emphasis  on  fleet  air  defense  and  fleet  air  superiority.  Within 
this  overall  mission  are  eight  mission  phases:  (1)  prelaunch,  (2)  take- 

off and  postlaunch,  (3)  in-flight  navigation,  (4)  surveillance,  (5)  combat, 
(6)  in-flight  fundamentals,  (7)  approach  and  landing,  and  (8)  post-mission. 

Mission  Elements 

Mission  elements,  which  are  specific  subsets  of  the  mission  phases, 
describe  more  detailed  functional  level  performance.  For  example,  "con- 
figure for  attack"  is  one  of  six  mission  elements  of  the  combat  mission 
phase. 

First-order  Task 

Each  mission  element  is  described  by  a series  of  first-order  tasks — 
cognitive  and  psychomotor  tasks  that  must  be  performed  by  the  aircrew  for 
mission  completion.  These  tasks  are  the  last  level  in  the  task  hierarchy 
for  which  a simple  description  of  the  job  may  apply  to  both  crew  members. 

For  instance,  both  the  pilot  and  NFO  must  reconfigure  the  Weapons  Control 
System  (WCS),  but  how  each  does  this  varies  greatly. 

Under  each  first-order  task  is  a detailed  listing  of  its  subtasks, 
broken  down  by  (1)  task  objective  or  goal,  (2)  task  determination,  or  the 
decisions  necessary  to  implement  the  task,  (3)  task  identification  by  code, 
and  (4)  task  implementation,  or  the  action  to  be  taken.  This  last  step, 
the  lowest  level  in  the  task  analysis,  describes  exactly  how  the  task  is 
to  be  accomplished  in  both  psychomotor  and  cognitive  terms.  Figure  2 is 
an  example  of  the  format  used  to  store  the  task  analysis  data. 

One  goal  of  the  task  analysis  documentation  was  to  provide  for 
computer  processing  of  all  task  data  if  and  when  a computer  became  avail- 
able. An  important  step  toward  this  goal  is  the  map  in  the  upper  right 
of  the  example  in  Figure  2.  Each  page  of  the  task  inventory  contains  such 
a map,  which  allows  the  user  of  the  document  to  immediately  locate  himself 
in  the  context  of  the  job  tasks  described. 


* 
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Hierarchies  and  Sequences 

Okey  (1971),  in  his  study  of  learning  hierarchies,  made  the  following 
observation: 

The  idea  that  learning  is  cumulative  or  hierarchial  is  a 
common-sense  notion.  Intuitively  one  feels  that  a learner 
must  master  arithmetic  before  tackling  algebra  or  must 
learn  to  distinguish  certain  letter  sounds  before  learn- 
ing to  read.  As  a result,  certain  school  courses  may  be 
listed  as  prerequisites  for  other  courses.  A group  of 
topics  may  be  ordered  in  sequence  within  courses  under 
the  assumption  that  one  topic  should  be  studied  before 
another.  Efforts  have  been  made  also  at  sequencing  in- 
struction on  a level  more  detailed  than  courses  or  topics 
within  a course.  The  aim  has  been  to  develop  sequences 
of  learning  tasks  at  a level  of  single  performance  out- 
comes. (p.  87) 

The  earliest  insights  into  optimum  sequencing  of  instruction  grew  out 
of  the  postwar  programmed  instruction  movement.  As  early  as  1954,  Skinner 
was  suggesting  the  logical  necessity  of  sequence.  Lumsdaine  (1964)  stated 
that  the  basic  assumption  of  any  programmed  instruction  is  that  there  is 
an  ideal,  or  optimal,  learning  sequence.  A review  of  the  role  of  sequence 
in  instruction  can  be  found  in  Niedermeyer  (1968),  who  suggested  that  se- 
quence may  not  be  as  important  as  has  been  first  thought. 

Empirical  support  for  the  logical  sequencing  of  instruction  is  ambiguous 
at  best.  Briggs  (1968)  attributed  this  more  to  experimental  design  limita- 
tions than  to  some  kind  of  learning  phenomenon.  The  primary  limitation  on 
instructional  sequence  research  has  been  the  brief  instructional  time  used 
in  most  studies.  The  few  studies  of  longer  learning  periods  have  produced 
positive  results  that  show  logical  sequence  does  enhance  learning  over  a 
reasonably  long  learning  period  (Payne,  Krathwohl,  & Gordon,  1967). 

Most  theoretical  work  in  instructional  hierarchies  has  been  directed 
toward  a taxonomy  of  hierarchical  levels  (Gagne,  1965,  1970)  and  in  spite 
of  ambiguous  empirical  support  and  the  embryonic  state  of  current  instruc- 
tional theory,  most  modern  instructional  technologists  view  hierarchies  as 
necessary  in  the  instructional  design  process  (e.g.,  Gleiner  & Lephene, 

1975). 

Researchers  in  the  present  effort  determined  that  a hierarchical  analysis 
of  job  tasks  in  the  task  inventory  would  (1)  lead  to  an  initial  sequencing 
of  the  training  course  and  (2)  identify  Individual  information  bits  that 
would  have  to  be  taught.  This  analysis  basically  worked  backwards  from 
the  job  task  to  a group  of  discrete  prerequisite  skill  and  knowledge  ele- 
ments. Figure  3 is  an  example  of  one  such  hierarchy. 

Each  discrete  element  of  the  hierarchy  was  transformed  into  a completely 
stated  learning  objective  with  specified  behavior,  conditions,  and  criteria. 
Each  of  these  learning  objectives,  therefore,  defined  one  test  point  of 
the  yet-to-be-developed  F-14  aircrew  training  curriculum. 
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PILOT  WILL  ESTABLISH  INITIAL 
SEARCH  CONFIGURATION 


Figure  3.  Example  of  task  analysis  hierarchy  (Veda,  Note  4). 


The  final  learning  objective  hierarchies  resulted  from  an  iterative 
review  process  from  the  contractor  to  VF-124  and  NAVPERSRANDCEN  personnel. 
The  final  set  of  learning  objectives  and  hierarchies  fill  20  volumes  with 
one  learning  objective  per  page  (Figure  4). 

A failing  of  most  major  instructional  design  efforts  is  their  failure 
to  document  the  end-products  of  the  design  process.  Without  complete  docu- 
mentation, linking  the  steps  in  the  development  and  maintaining  the  context 
of  any  specific  instructional  event,  continuity  is  lost  and  the  resulting 
training  program  will  fall  far  short  of  what  should  be  expected  from  ISD. 

The  present  documentation  of  the  F-14  learning  objectives  avoids  this 
pitfall  by  describing  in  detail  where  each  learning  objective  in  the  train- 
ing program  fits  and  how  retrieval  of  instructional  information  and  assump- 
tions can  be  accomplished  either  manually  or  with  data  processing  assistance. 


INSTRUCTIONAL  DECISION  MAKING 

Job  Entry  Standards 

As  Che  task  analysis  effort  was  being  completed  and  the  objectives 
and  their  hierarchies  were  being  developed,  fighter  squadrons  (VF-1  and 
VF-2)  returned  from  the  F-14's  first  deployment.  A sample  of  the  pilots 
and  NFOs  was  given  the  task  inventories  and  asked  to  rate  the  importance 
of  each  task  on  the  following  seven-point  scale: 

1.  Course  graduate  should  be  able  to  perform  this  task  as  rapidly 
and  accurately  as  an  experienced  F-14  pilot/NFO. 


2.  Course  graduate  should  be  able  to  perform  this  task  as  accurately 
but  not  as  rapidly  as  an  experienced  F-14  pilot/NFO. 

3.  Course  graduate  should  be  able  to  perform  this  task  as  rapidly 
but  not  as  accurately  as  an  experienced  F-14  pilot/NFO. 

4.  Course  graduate  should  be  able  to  perform  this  task  but  with 
neither  the  speed  nor  the  accuracy  of  an  experienced  F-14  pilot/NFO. 

5.  Course  graduate  should  have  in-depth  knowledge  about  this  task 
but  not  be  able  to  perform  it. 

6.  Course  graduate  should  know  about  this  task  but  not  be  able  to 
perform  it. 

7.  Course  graduate  should  neither  know  about  nor  be  able  to  perform 
this  task. 

Table  2 presents  the  distribution  of  ratings  by  four  raters  for  the 
tasks  in  the  surveillance  and  combat  mission  phases  of  the  task  inventory. 

Five  major  findings  that  influenced  later  instructional  decisions  were: 


A 


1.  Learning  objective  standards  should  emphasize  accuracy  rather  than 
speed. 

2.  Approximately  15  percent  of  the  tasks  contained  within  the  sur- 
veillance and  combat  mission  phases  require  the  highest  level  of  attention 
(i.e.,  were  consistently  rated  number  one). 

3.  Graduating  aircrews  are  expected  to  perform  nearly  99  percent  of 
the  tasks  in  the  inventory.  However,  slightly  less  than  half  of  these 
tasks  were  rated  1,  2,  or  3. 

4.  The  surveillance  mission  phase  is  the  most  critical  tactical  area 
for  the  NFO. 

5.  The  combat  mission  phase  la  the  most  critical  tactical  area  for 
the  pilot. 
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Table  2 


Ratings  of  the  Relative  Importance  of  F-14  Combat  and  Surveillance  Tasks 


Scale  Category 

Percentage  of  Tasks  per  Scale  Category 

Surveillance  Phase 

Combat  Phase 

Pilot  Tasks 

N - 298 

N - 888 

1 

15 

15 

2 

9 

24 

3 

1 

0 

4 

70 

59 

5 

0 

1 

6 

5 

1 

7 

0 

0 

TOTAL 

100 

100 

NF0  Tasks 

N - 1204 

N - 1024 

1 

14 

17 

2 

50 

39 

3 

0 

1 

4 

36 

43 

5 

0 

0 

6 

0 

0 

7 

0 

0 

TOTAL 

100 

100 

Note.  Data  taken  from  Veda,  Note  3. 


These  data  were  Invaluable  in  prescribing  the  Initial  limits  of  the 
ISD.  Although  very  few  tasks  were  dropped  from  training  consideration, 
initial  planning  could  (1)  designate  tasks  that  needed  intensive  training 
and  (2)  prescribe  the  criteria  dimensions  of  the  others  (i.e.,  time,  accuracy, 
or  number  of  correct  responses).  These  data  gave  the  instructional  designers 
the  real-world  requirements  base  from  which  they  could  construct  their 
training  program. 

Learning  Types 

The  problem  of  defining  types  of  learning  is  a subset  of  the  larger 
problem  of  deriving  usable  behavioral  taxonomies,  which  will  be  discussed 
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in  some  detail  in  the  following  section.  However,  if  there  is  a finite, 
manageable  number  of  learning  types  and  we  can  discover  how  best  to  accom- 
plish each  type,  instructional  theory  can  progress. 


Early  learning  research  and  theory  were  centered  on  reducing  all  learning 
to  some  single  phenomenon.  Thorndike  was  Interested  in  associations  and 
Pavlov,  in  reflexes.  Ebbinghaus  did  his  major  work  on  verbal  learning, 

Kohler  concentrated  on  problem  solving,  and  Hull  hypothesized  complex 
stimulus-response  chains.  Although  fascinating,  their  work  has  been  of 
minimal  help  to  the  classroom  instructor  or  the  designer  of  instructional 
programs . 

Gagne  (1965,  1970)  defined  eight  kinds  of  learning  that  are  related 
hierarchically:  problem  solving,  rule  learning,  concept  learning,  discrim- 

ination learning,  verbal  association,  chaining,  stimulus-response  learning, 
and  signal  learning.  (For  a succinct  discussion  of  them,  see  Gagne,  1971.) 
Although  Gagne  was  not  the  first  theoretician  to  identify  different  types 
of  learning,  he  was  the  first  to  bridge  the  gap  between  learning  theory 
and  instructional  theory.  He  does  not  postulate  completely  different 
learning  mechanisms,  but  "merely  observed  that  learning  which  occurs  under 
different  conditions  has  different  characteristics  and  can  consequently 
be  considered  different  as  far  as  instruction  is  concerned"  (Merrill, 

1971b,  p.  34). 

Merrill  and  Tennyson  (1971),  after  comparing  Bloom  (1956),  Gagne  (1965), 
and  Merrill  (1971b),  concluded  that  the  10  categories  of  learned  behavior 
described  by  Merrill  comprise  a superior  learning  taxononlc  model  (i.e., 
comes  closest  to  describing  distinct  and  observable  learning  types) . Figure 
5 shows  Merrill's  (1971a)  10  categories  of  learned  behavior. 


Emotional 

— 

Emotional 
(Signal  Learning) 

Psychomotor 

Topographic 

Chaining 

Complex  Skill 

Memorization 

Naming 

Serial  Memory 

Discrete  Memory 

Complex 

Cognitive 

Classification 

_ 

Analysis 

Problem  Solving 

Figure  5.  Merrill's  learning  types. 
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Gagne's  original  eight  learning  types  required  a strict  hierarchy  of 
types — stimulus-response  preceded  chaining,  chaining  preceded  discrimina- 
tion, etc.  Merrill's  paradigm  defines  four  learning  constructs — emotional, 
psychomotor,  memorization,  and  complex  cognitive — the  latter  three  of  which 
have  self-contained  hierarchies.  In  other  words,  there  is  an  order  to 
learning  psychophysical  tasks— from  topographic  (stimulus-response)  to  com- 
plex skill.  However,  these  are  not,  in  turn,  necessarily  prerequisite  to 
memorization  or  complex  cognitive  skills. 

In  an  interesting  progression,  Markle  and  Tiemann  (1973)  have  further 
developed  Merrill's  model.  Essentially,  the  Markle  and  Tiemann  model 
(Figure  6)  adds  a third  dimension  to  the  Merrill  model  and  employs  a more 
useful  vocabulary.  For  example,  the  idea  of  algorithmic  sequences  is 
immediately  understandable  to  the  operator  of  a state-of-the-art  weapon 
control  system.  Likewise,  the  concept  of  kinesthetic  repertory  is  quickly 
grasped  by  the  pilot  of  a high-performance  aircraft. 


PSYCHOMOTOR 


MEMORY 


COMPLEX 

COGNITIVE 
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KINESTHETIC 

REPERTOIRES 


CHAINS 


RESPONSES 


1 


VERBAL 

REPERTOIRES 


STRATEGIES 


SEQUENCES 


SERIAL  . ALGORITHMS 
MEMORY  . 


PRINCIPLES 
(RULE  APPLYING) 


PAIRED  / MULTIPLE 
ASSOCIATES  j DISCRIMINATIONS  j 

ASSOCIATIONS 


CONCEPTS 


EMOTIONAL 


Figure  6.  Markle 's  and  Tiemann 's  learning  types. 


The  next  step,  once  the  decision  was  made  to  use  the  Markle  and  Tiemann 
model,  was  to  bridge  the  gulf  between  the  pilot  and  NFO  behaviors  in  the  air- 
craft and  the  more  abstract  learning  types. 
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Ever  since  Bloom  (1956)  published  his  Taxonomy  of  Educational  Objectives 
instructional  theoreticians  and  researchers  have  carried  on  the  search  for 
some  way  to  classify  human  behavior  into  discrete  categories.  If  a set  of 
behaviors  falls  into  a particular  category,  then  research  into  training  of 
that  particular  category  can  yield  usable  generalizations  with  real  world 
applications.  It  has  been  20  years  since  Bloom  first  published  his  cogni- 
tive domain  taxonomy  and  the  goal  of  a generalizable  taxonomy  remains  as 
elusive  as  ever. 


A taxonomy  implies  that  man's  behavior — at  least  the  part  that  is  of 
concern  to  the  trainer — is  discrete;  that  is,  it  can  fit  into  some  finite 
number  of  classifications.  Invariably,  however,  taxonomies  of  behavior 
tend  to  be  situationally  specific  (e.g..  Miller,  1969)  or  directed  toward 
a manageable  domain,  like  psychomotor  (e.g.,  West,  1957)  or  cognitive  (e.g 
Bloom,  1956).  Looking  for  the  ultimate  behavior  taxonomy  is  like  looking 
for  the  ultimate  hammer — it  is  better  to  let  the  task  determine  the  tool. 


In  reducing  the  jobs  of  the  F-14  aircrew  into  discrete  tasks,  a con- 
tinuum of  job  task  behavior  was  perceived.  At  one  end  is  a group  of  tasks 
that  are  singular  responses  to  specific  system  stimuli;  and  at  the  other 
end,  tasks  requiring  unique  responses  to  the  aircraft's  environment.  The 
former  have  been  designated  "reactive  behavior"  and  the  latter,  "decision 
behavior."  The  "reactive"  end  of  the  scale  is  very  functional  and  emphasizes 
the  machine  system  stimulus.  The  "decision"  end  of  the  scale  is  highly 
cognitive  and  emphasizes  the  performing  human. 


Table  3 shows  the  task  behavior  scale  and  its  seven  discrete  cate- 
gories; Figure  7,  the  continuous  scale  from  which  the  categories  are  de- 
rived; and  Figure  8,  an  algorithm  that  was  developed  to  assign  tasks  into 
appropriate  categories. 


Although  the  influence  of  Gagne  (1970)  on  the  categories  is  apparent, 
it  must  be  emphasized  that  these  operator  behavior  categories  are  not 
analogous  to  Gagne's  (or  anyone  else's)  learning  types.  The  categories 
describe  system  operator  behavior.  The  type  of  learning  that  must 
occur  to  acquire  the  behavior  follows  from  the  categorization  but  there 
is  not  a one  to  one  relationship. 


The  behavioral  category  scale  is  used  twice  in  the  instructional  de- 
cision-making process.  First,  it  can  lead  one  from  the  operator's  behavior- 
which  is  categorized  first — to  a designation  of  learning  type.  Table  4 
shows  the  relationship  of  the  behavioral  categories  to  the  models  of  Gagne, 
Merrill,  and  Markle  and  Tlemann.  That  of  Markle  and  Tiemann  was  selected 
as  the  best  decision  aid  to  this  instructional  design  effort  because  its 
language  and  organization  most  nearly  relate  to  the  operator  repertoire  of 
required  skills.  The  point  of  getting  to  the  learning  types  is  to  be  able 
to  draw  upon  whatever  R&D  data  base  may  exist  that  would  indicate  how  best 
to  train  a particular  learning  type.  Since  there  is  a hierarchical  rela- 
tionship to  the  learning  types,  instructional  sequencing  is  also  facilitated 


3Much  of  this  section  was  presented  in  Brock  (1976) 
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Operator  Task  Behavior  Categories 
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Type 

Name 

Description 

1 

Stimulus  Response 

Specific  operator  response  to 
a system-generated  stimulus. 

2 

Sequenced  Chaining 

Specific  set  of  operator  re- 
sponses initiated  by  a system 
stimulus. 

3 

Algorithmic  Chaining 

Set  of  operator  responses 
initiated  by  a system’s 
stimuli;  each  response  may 
act  as  a stimulus  for  further 
responses  in  the  chain 

4 

Interactive  Chaining 

Operator  ferrets  out  sys:em 
and  environmental  stimuli  and 
makes  set  of  responses. 

5 

Multiple  Discrimination 

Operator  discriminates  among 
system  and  environmental 
stimuli. 

6 

Concept  and  Rule  Application 

Operator  applies  concepts  and 
rules  to  system  and  environ- 
mental stimuli. 

7 

Decision  Making 

Operator  solves  unique 
problems  in  the  system  and 
environment. 

Note. 

From  Brock  (1976). 
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BEHAVIORIAL  CATEGORIES 


REACTIVE 

BEHAVIOR 


DECISIVE 

BEHAVIOR 


FUNCTIONAL  - COGNITIVE 


MACHINE  - HUMAN 

Each  Cask  nay  be  described  by  the  following  behavior: 

1.  A specific  operator  response  to  a system-generated  stimulus. 

2.  A specific  set  of  operator  responses  initiated  by  a system  stimulus. 

3.  A set  of  operator  responses  initiated  by  a system's  stimuli. 

4.  A situation  in  which  the  operator  ferrets  out  system  and  environmental 
stimuli  and  makes  sets  of  responses. 

5.  A situation  in  which  the  operator  discriminates  among  system  and 
environmental  stimuli. 

6.  A situation  in  which  the  operator  applies  concepts  and  rules  to 
system  and  environmental  stimulus  situations. 

7.  A situation  in  which  the  operator  solves  unique  problems  with 
system  and  environmental  stimuli. 


Figure  7.  Behavioral  category  for  F-14  weapons 
system  operation  (Brock,  1976). 
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The  second  use  made  of  che  task  category  scheme  is  In  the  assignment 
of  practice  and  test  characteristics.  A high-fidelity  simulator  is  not 
necessarily  the  best  place — or  even  an  adequate  place — to  practice  or  measure 
some  kinds  of  task  performance.  Behavioral  categories  6 and  7 appear  to  be 
best  practiced  and  tested  in  a classroom  or  learning  carrel  which  mini- 
mize Irrelevant  cues.  For  example,  solving  an  intercept  geometry  problem 
is  a highly  cognitive  skill  that  requires  much  practice.  To  hav»  to  develop 
finesse  at  this  skill  while  "flying"  a simulator  (with  all  its  attendant 
problems)  is  unreasonable.  This  type  of  classroom  cognitive  practice  and 
test  process  has  been  successful  with  surface  Navy  officers  (McCutcheon  & 
Brock,  1971;  Brock,  1972). 

The  tasks  in  categories  4 and  5 are  those  that  require  the  high  fidelity 
of  a Weapon  System  Trainer  for  task  practice  and  test  (e.g.,  flight  maneuvers 
for  the  pilot  and  complex  search  procedures  for  the  NFO). 

Categories  1,  2,  and  3 will  be  best  practiced  in  dynamic  part-task 
trainers  that  only  need  to  simulate  key  subsystem  cues  and  responses.  The 
kind  of  part-task  training  developed  by  Crawford  (1976)  for  the  S-3A  Viking 
aircraft  allows  efficient  practice  of  skills  in  these  categories. 

The  behavioral  category  system,  then,  has  served  as  a bridge  between 
the  operational  system  and  the  training  system  and  has  stimulated  the  devel- 
opment of  training  strategy  decision  aids. 

Sequencing 

The  large-scale  sequence  of  the  training  course  is  based  on  nothing 
more  sophisticated  than  "fundamental  before  complex,  single  task  before  dual 
task."  The  Markle  and  Tiemann  (1973)  model  becomes  critical  at  the  indivi- 
dual learning  unit  level  for  it  is  at  this  level  that  necessary  psycholog- 
ical prerequisites  to  learning  become  critical. 

The  task  analysis  hierarchies  described  earlier  (Figure  3)  become  the 
intraunit  sequencing  outline.  The  prescribed  order  of  each  hierarchy  is 
the  Initial  sequence  of  the  course.  However,  many  of  the  high-level  F-14 
aircrew  learning  objectives  have  common  prerequisite  skills.  Finding  these 
common  prerequisites  defined  the  low-level  learning  objectives  that  make 
up  either  pretest  components  or  preliminary  instructional  modules.  (The 
assumption  that  the  skills  that  appear  prerequisite  are,  in  fact,  prerequi- 
site has  not  been  tested;  the  consequences  of  a pilot  or  NFO  not  learning 
a critical  skill  are  far  too  serious  for  such  experimentation.)  An  algorithm 
developed  and  used  by  the  ISD  team  for  its  intraunit  sequencing  decisions 
is  shorn  in  Figure  9. 

To  date,  no  other  cases  in  which  a learning  type  model  such  as  Markle's 
and  Tiemann 's  was  used  on  a scale  approaching  that  described  herein  have 
been  found. 
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Media  Selection 

The  problem  of  selecting  the  best  instructional  medium  to  maximize 
the  acquisition  of  some  specific  skill  or  knowledge  is  remarkably  intrac- 
able; the  search  for  a general-purpose  media  selection  model  parallels  the 
previously  discussed  search  for  the  ultimate  ISD  model.  The  most  recent 
Navy  media  selection  guide  frankly  states  that  it  must  be  used  by  a know- 
ledgeable instructional  specialist  (Braby,  Henry,  Parrish,  & Swope,  1975; 
Aagard  & Braby,  1976).  Sugarman,  Buckenmaeir  and  Johnson  (1975)  state  that 
a complete  catalogue  linking  training  problems  to  media  solutions  cannot 
be  workable  because  the  necessary  information  simply  is  not  available. 

The  suggestion  is  that  there  are  an  infinte  number  of  training  problems 
and  an  infinite  number  of  media  solutions.  Thus,  the  ISD  team  decided  to 
use  its  own  finite  number  of  training  problems  and  the  practical  and  acces- 
sible media  solutions  to  build  a problem-specific  media  selection  algorithm. 
An  example  from  this  algorithm  is  shown  in  Figure  10. 4 

With  the  exception  of  computer-assisted  instruction  (CAI),  all  the  media 
selections  are  based  on  current  VF-124  capabilities.  CAI  was  added  because 
it  describes  the  optimum  characteristics  for  specific  training  needs.  It 
allows  for  a dynamic,  interactive,  and  adaptive  training  system  at  a level 
unavailable  through  conventional  instructional  media.  If  CAI  does  not  be- 
come a practical  option  to  VF-124,  alternatives  are  provided  in  the  al- 
gorithm. 5 

Practice  and  Test  Events 

Up  to  now,  this  report  has  discussed  either  historical  or  ongoing  pro- 
cesses. This  brief  section  will  discuss  the  Implications  of  these  processes 
for  the  design  of  practice  and  test  exercises. 

There  is  considerable  evidence  that  high-fidelity  simulation  (i.e., 
exact  reproduction  of  real  world  characteristics)  is  not  necessary  for 
the  acquisition  of  all  complex  psychomotor  and  cognitive  skills  (Prophet  & 
Boyd,  1970).  In  fact,  Crawford  (1976)  found  that  a part-task  trainer  that 
had  at  least  one  major  difference  from  the  system  being  simulated  trained 
better  than  training  conducted  on  the  actual  system. 


4Complete  copies  of  this  algorithm  may  be  obtained  from  Veda,  Inc. 

5 Although  many  CAI  systems  are  currently  available  that  differ  signi- 
ficantly from  one  another,  the  term  is  used  here  generically  to  describe 
an  instructional  delivery  system  with  the  characteristics  described  in  the 
text. 
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le  of  media  selection  algorithm  (Veda,  Note  5;  reprinted  with  permission). 


The  behavioral  category  model  described  in  this  report  and  in  Brock  (1976) 
can  be  used  to  identify  both  the  required  degree  of  simulation  and  the  spe- 
cific factors  to  be  simulated.  For  instance,  a task  rated  7,  "Operator 
solves  unique  problems  in  the  system  and  environment,"  is  the  kind  of  task 
for  which  the  student  must  practice  a series  of  problems  with  infinite 
dimensions.  In  other  words,  what  needs  to  be  simulated  is  not  hardware 
but  situations.  Determining  what  course  and  speed  to  fly  to  fire  a par- 
ticular kind  of  weapon  requires  high-level  problem-solving  under  extremely 
limiting  time  restraints.  Flying  that  course  and  speed — a 4 in  the  behavior 
scale — does  require  a multidimensional  simulator.6 


Some  tasks  are  best  practiced  on  part-task  trainers  that  can  simulate 
the  specific  cue  and  response  variables  needed  for  skill  acquisition. 
Crawford  (1976)  and  Crawford,  Hurlock,  Padilla,  and  Sassano  (1976)  have 
described  an  S-3A  trainer  from  which  students  learned  specific  procedures 
and  then  went  on  to  the  actual  equipment  in  a high-fidelity  simulator. 
These  students  performed  better  in  terms  of  both  the  number  of  problems 
solved  and  time-to-criterion  than  those  students  who  had  practiced  exclu- 
sively in  the  high-fidelity  simulator. 


A key  factor  in  this  discussion  is  the  nature  of  tests  that  evolve  from 
these  acquisition  and  practice  exercises.  If  the  learning  and  practice  of 
problem-solving  skills  is  best  conducted  in  a classroom  or  learning  carrel, 
then  so  is  testing.  In  fact,  the  ideal  test  is  to  have  the  student  practice 
each  skill  until  he  achieves  the  criterion.  If  real-world  constraints  pre- 
vent this,  at  least  the  student  should  be  tested  along  the  same  dimensions  he 
has  been  practicing. 


Recent  experiences  (Moore,  Note  6)  have  indicated  that  there  are  at 
least  three  stages  in  the  complete  acquisition  of  complex  psychomotor  and 
cognitive  skills:  (1)  the  acquisition  phase,  in  which  time  and  probably 

accuracy  criteria  are  low,  (2)  the  development  of  skill  proficiency  before 
it  is  integrated  into  the  total  job  of  the  operator,  and  (3)  the  integra- 
tion of  the  newly  acquired  skill  into  the  overall  behavioral  pattern  required 
to  operate  the  system.  At  least  one  study  has  indicated  that  there  may  be 
a fourth  level  (or  even  more) — at  least  in  the  cognitive  domain — that  is 
reached  only  after  extended  operating  experiences  (Eddowes  & Waters,  1976). 
Early  evidence  suggests  that  the  behavioral  category  scale  discussed  earlier 
may  only  be  applicable  in  stage  one  learning  (the  initial  acquisition  phase), 
at  least  as  far  as  determining  the  degree  of  simulation  required. 


6 That  a high  degree  of  simulation  might  not  be  needed  even  here  is  a 
very  real  issue,  but  one  that  goes  beyond  the  scope  of  this  paper.  For 
the  relevant  arguments,  the  reader  is  referred  to  Prophet  and  Boyd  (1970) 
or  Provenmlre  and  Roscoe  (1971). 
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DISCUSSION 


Much  of  this  research  was  simply  a review  of  Instructional  technology 
state-of-the-art.  However,  the  specific  algorithms  and  the  behavioral 
category  scale  represent  Innovations  that  should  have  Immediate  applica- 
tion to  various  and  dissimilar  training  programs.  Some  evidence  for  this 
conclusion  has  been  indicated  (DeLong,  Note  7). 

This  process  is  not  the  Idealized  and  elusive  fully  proceduralized 
instructional  design  system.  Many  of  the  techniques  developed  in  this 
effort  came  of  necessity,  and  have  been  validated  only  to  the  extent  that 
they  allowed  the  process  to  continue.  An  efficient  job  relevant  training 
program  is  being  designed,  but  whether  or  not  it  will  ever  be  conducted 
remains  to  be  seen. 

The  typical  ISD  team  is  composed  of  Job  experts  and  Instructional 
specialists.  In  the  military,  the  former  role  is  taken  by  uniformed  per- 
sonnel and  the  latter  by  staff  education  specialists.  However,  In  the 
reported  effort,  the  contractor  functioned  initially  as  Job  expert  and 
eventually  as  Job  expert,  instructional  designer,  and  documentation  pro- 
ducer; NAVPERSRANDCEN  personnel  served  as  program  consultants.  The  F-14 
training  community's  time  is  completely  dedicated  to  ongoing  training 
requirements.  An  effort  of  this  magnitude  is  simply  beyond  the  manpower 
resources  of  an  operational  unit.  VF-124  did  have  the  authority  to  review 
and  approve  each  step  of  the  process;  all  steps  reported  herein  have  been 
so  approved.  Although  command  lines  were  occasionally  fuzzy  and  some  time 
was  lost  in  the  review  process  used  in  the  present  approach,  it  seems  a 
reasonable  approach  to  ongoing  training  programs  requiring  major  ISD  with- 
out a concurrent  reduction  in  student  load. 

COMNAVAIRSY SCOM  had  planned  that  this  ISD  program  would  coincide  with 
a computer-managed  training  (CMT)  system  being  developed  as  a separate 
effort,  but  that  CMT  program  failed  and  is  no  longer  an  available  op- 
tion. However,  the  Versatile  Training  System,  which  has  previously  proved 
succesaful  in  managing  an  aircraft  maintenance  training  program  Is  being 
Installed  at  VF-124.  Because  of  the  documentation  and  coding  system  used 
in  the  ISD  effort,  full  compatibility  between  the  new  system  and  the  new 
eyllabl  is  foreseen. 

An  extensive  review  of  the  current  instructional  technology  state-of- 
the-art  has  produced  few  hard  findings,  but  rather  a number  of  significant 
guidelines  for  the  designer  of  instruction,  be  he  behavioral  scientist  or 
Navy  Chief  Petty  Officer. 


29 


CONCLUSIONS 

1.  To  stop  designing  training  until  a definitive  ISD  model  is  developed 
is  out  of  the  question;  there  may  never  be  such  a model. 

2.  The  F-14  ISD  team  has  proceeded  from  a task  analysis  to  a syllabus 
outline  and  the  evidence  indicates  that  an  effective  and  efficient  training 
program  is  being  developed. 

3.  The  documentation  produced  by  the  ISD  contractor  will  allow  rapid 
Integration  of  the  new  F-14  training  syllabi  into  the  Navy's  Versatile 
Training  System. 


RECOMMENDATIONS 

1.  The  F-14  aircrew  training  syllabi  developed  as  described  herein 
should  be  implemented  with  minimum  interference  with  ongoing  training. 

2.  The  algorithms  and  decision  aids  developed  on  this  project  should 
become  part  of  the  Navy's  available  instructional  technology  resources. 

3.  The  possibility  of  using  part-task  trainers  in  the  F-14  training 
program  should  be  thoroughly  investigated.  The  potential  cost  savings 
are  well  worth  committing  seed  money  to  such  a program. 

4.  The  GAI  options  resulting  from  the  media  selection  algorithms  should 
be  carefully  examined. 

5.  Begin  an  R&D  program  to  study  learning  hierarchies.  The  assump- 
tion that  the  logically  derived  hierarchies  are  necessary  to  the  learning 
process  should  be  investigated  under  other  than  operational  conditions, 
where  the  implications  of  a task  not  being  learned  are  minimal. 
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